Feature: add activate-launchplan command to pyflyte #1588
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wckdman
requested review from
wild-endeavor,
kumare3,
eapolinario,
pingsutw and
cosmicBboy
as code owners
|
Thank you for opening this pull request! 🙌 These tips will help get your PR across the finish line:
|
wckdman
force-pushed
the
feature/pyflyte-activate-launchplan
branch
from
093ed08 to
80013ec
Compare
Codecov Report
@@ Coverage Diff @@
## master #1588 +/- ##
==========================================
+ Coverage 69.96% 70.00% +0.03%
==========================================
Files 325 327 +2
Lines 29807 29843 +36
Branches 5364 5376 +12
==========================================
+ Hits 20854 20891 +37
+ Misses 8430 8428 -2
- Partials 523 524 +1
... and 1 file with indirect coverage changes Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
wckdman
force-pushed
the
feature/pyflyte-activate-launchplan
branch
from
69f7b55 to
3c30baa
Compare
pingsutw
reviewed
Apr 13, 2023
| """ | ||
|
|
||
|
|
||
| @click.command("activate-launchplan", help=_activate_launchplan_help) |
There was a problem hiding this comment.
could we change the command to launchplan? like
pyflyte launchplan LP --activate/deactivateThere was a problem hiding this comment.
Totally agree!
Shall I set a default value for this action (activate/deactivate) or make it required? WDYT?
pingsutw
reviewed
Apr 15, 2023
Signed-off-by: Artem Petrov <[email protected]>
Signed-off-by: Artem Petrov <[email protected]>
Signed-off-by: Artem Petrov <[email protected]>
Signed-off-by: Artem Petrov <[email protected]>
wckdman
force-pushed
the
feature/pyflyte-activate-launchplan
branch
from
2bdd426 to
c534416
Compare
|
@pingsutw done ✅ |
pingsutw
approved these changes
Apr 17, 2023
kumare3
approved these changes
Apr 17, 2023
|
@kumare3 it looks like a recurring problem according to "actions" tab. even the build at master branch has failed. |
|
Test is failing in all the PRs for some reason, I'm fixing it today. |
pingsutw
approved these changes
Apr 18, 2023
|
Congrats on merging your first pull request! 🎉 |
fg91
pushed a commit
that referenced
this pull request
Apr 23, 2023
* add activate-launchplan command to pyflyte Signed-off-by: Artem Petrov <[email protected]> * fix linting Signed-off-by: Artem Petrov <[email protected]> * rework launchplan command Signed-off-by: Artem Petrov <[email protected]> * fix linting Signed-off-by: Artem Petrov <[email protected]> --------- Signed-off-by: Artem Petrov <[email protected]> Signed-off-by: Fabio Graetz <[email protected]>
eapolinario
pushed a commit
that referenced
this pull request
May 16, 2023
* add activate-launchplan command to pyflyte Signed-off-by: Artem Petrov <[email protected]> * fix linting Signed-off-by: Artem Petrov <[email protected]> * rework launchplan command Signed-off-by: Artem Petrov <[email protected]> * fix linting Signed-off-by: Artem Petrov <[email protected]> --------- Signed-off-by: Artem Petrov <[email protected]>
eapolinario
added a commit
that referenced
this pull request
May 17, 2023
* Data subsystem (#1526) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> * Pluck retry from flytekit and into sagemaker (#1411) * Remove retry from flytekit's setup.py and regenerate requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Add to sagemaker Signed-off-by: Eduardo Apolinario <[email protected]> * Remove retry from sagemaker plugin requirements file Signed-off-by: Eduardo Apolinario <[email protected]> * Restore doc-requirements.txt Signed-off-by: eduardo apolinario <[email protected]> * Fix bad merge Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Update the pypi wait (#1554) Signed-off-by: Yee Hing Tong <[email protected]> * Stream Directories and Files using Flyte (#1512) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> * Make `FlyteFile` compatible with `Annotated[..., HashMethod]` (#1544) * fix: Make FlyteFile compatible with Annotated[..., HashMethod] See issue #3424 Signed-off-by: Adrian Rumpold <[email protected]> * tests: Add test case for FlyteFile with HashMethod annotation Issue: #3424 Signed-off-by: Adrian Rumpold <[email protected]> * fix: Use typing_extensions.Annotated for py3.8 compatibility Issue: #3424 Signed-off-by: Adrian Rumpold <[email protected]> * fix: Use `get_args` and `get_origin` from typing_extensions for py3.8 compatibility Issue: #3424 Signed-off-by: Adrian Rumpold <[email protected]> * fix(tests): Use fixture for local dummy file Signed-off-by: Adrian Rumpold <[email protected]> --------- Signed-off-by: Adrian Rumpold <[email protected]> * move FlyteSchema deprecation warning to initialization method (#1558) * move FlyteSchema deprecation warning to initialization method Signed-off-by: Niels Bantilan <[email protected]> * update Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> * add pod_template and pod_template_name arguments for ContainerTask (#1515) * add pod_template and pod_template_name arguments for ContainerTask Signed-off-by: Felix Ruess <[email protected]> * factor out _serialize_pod_spec into separate util function Signed-off-by: Felix Ruess <[email protected]> * model file changes, couple other changes Signed-off-by: Yee Hing Tong <[email protected]> * minor cleanup Signed-off-by: Felix Ruess <[email protected]> * add unit test for container_task pod_template Signed-off-by: Felix Ruess <[email protected]> * bump min version of flyteidl to 1.3.12 for pod template data config support Signed-off-by: Felix Ruess <[email protected]> * require flyteidl==1.3.12 in doc-requirements.txt Signed-off-by: Felix Ruess <[email protected]> --------- Signed-off-by: Felix Ruess <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * Pass locally defined scopes to RemoteClientConfigStore (#1553) Signed-off-by: franco-bocci <[email protected]> * TypeTransformer for TensorFlow model (#1562) * TypeTransformer for TensorFlow model Signed-off-by: Samhita Alla <[email protected]> * clean up Signed-off-by: Samhita Alla <[email protected]> * clean up Signed-off-by: Samhita Alla <[email protected]> * fix lint Signed-off-by: Samhita Alla <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]> * Remove fsspec flytekit plugin from main Dockerfile (#1561) Signed-off-by: Yee Hing Tong <[email protected]> * Device auth flow / Headless auth (#1552) * Device auth flow Signed-off-by: Ketan Umare <[email protected]> * Device AuthFlow is now available in flytekit Signed-off-by: Ketan Umare <[email protected]> * unit tests Signed-off-by: Ketan Umare <[email protected]> * test added Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * Fixed test Signed-off-by: Ketan Umare <[email protected]> * Fixed unit test Signed-off-by: Ketan Umare <[email protected]> * Fixed lint errors Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> * url encode secret in client credentials flow (#1566) * url encode secret first Signed-off-by: Yee Hing Tong <[email protected]> * nit Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> * Python run multiple files (#1559) * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * General Partial support in flytekit and multi-list support in flytekit (#1556) Signed-off-by: Ketan Umare <[email protected]> * fix: Silence keyring warnings by changing to debug (#1568) Co-authored-by: ggydush-fn <[email protected]> * Support GCP secrets (#1571) Signed-off-by: Yee Hing Tong <[email protected]> * Automatically remove unused import (#1574) * Automatically remove unused import Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Disallow partial lists in map tasks (#1577) * Disallow partial lists in map tasks Signed-off-by: eduardo apolinario <[email protected]> * Lint Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Remove duplicate reporting logic (#1578) Signed-off-by: Yee Hing Tong <[email protected]> * [Core feature] Convert List[Any] to a single pickle file (#1535) * Convert List[Any] to a single pickle file * remove redundant code * keep batchSize only if type contain flytePickle * fix error * add batch support to translate_inputs_to_literals * add ci test Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve comment Signed-off-by: Yicheng-Lu-llll <[email protected]> * add more ci test Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * handle HashMethod case Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve format Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * add test_is_batchable Signed-off-by: Yicheng-Lu-llll <[email protected]> * Add BatchSize Signed-off-by: Kevin Su <[email protected]> * test_batch_size Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Yicheng-Lu-llll <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Kevin Su <[email protected]> * Improve authoring structure documentation (#1572) Signed-off-by: Samhita Alla <[email protected]> * Unify sqlalchemy Dockerfiles (#1585) * Unify sqlalchemy Dockerfiles Signed-off-by: eduardo apolinario <[email protected]> * Simplifyte sqlalchemy Dockerfile Signed-off-by: eduardo apolinario <[email protected]> * Set pythonpath to /root Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Pyflyte build imageSpec (#1555) * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * pyflyte build Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * Support serialize and package Signed-off-by: Kevin Su <[email protected]> * more tests Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * move to plugin Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fixed tested Signed-off-by: Kevin Su <[email protected]> * fixed tested Signed-off-by: Kevin Su <[email protected]> * more tests Signed-off-by: Kevin Su <[email protected]> * Add support passing yaml in pyflyte run Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Feature/3506: Improve Type Conversion errors, use rich to prettify error messages (#1582) * improve input type conversion error Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> * fix Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add rich Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> * remove prototyping script, update loggers Signed-off-by: Niels Bantilan <[email protected]> * update __init__.py Signed-off-by: Niels Bantilan <[email protected]> * update logger Signed-off-by: Niels Bantilan <[email protected]> * update logger Signed-off-by: Niels Bantilan <[email protected]> * fix GE and pandera tests Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> * Add support nested FlyteFile in pyflyte run (#1587) * Add support nested FlyteFile in pyflyte run Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Set cache_regions=True in the case of s3fs (#1592) * Set cache_regions=True in the case of s3fs * Linting Signed-off-by: eduardo apolinario <[email protected]> * Fix tests Signed-off-by: eduardo apolinario <[email protected]> * More linting Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Set default ComSpec when running on Windows (#1595) * set default ComSpec when running on Windows Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Feature: add activate-launchplan command to pyflyte (#1588) * add activate-launchplan command to pyflyte Signed-off-by: Artem Petrov <[email protected]> * fix linting Signed-off-by: Artem Petrov <[email protected]> * rework launchplan command Signed-off-by: Artem Petrov <[email protected]> * fix linting Signed-off-by: Artem Petrov <[email protected]> --------- Signed-off-by: Artem Petrov <[email protected]> * nit (#1596) Signed-off-by: Yee Hing Tong <[email protected]> * Add log streaming to papermill plugin (#1129) * checkpoint Signed-off-by: Mike Zhong <[email protected]> * Experimental implementation works well. Instead of messing with the class, we utilize the interpolation and context to dynamically generate a wrapper around the desired script. In the wrapper, we cd to the ctx.working_directory, export the env variables (handled by an additional method to convert dict to str), and then pass the arguments to the script Signed-off-by: Mike Zhong <[email protected]> * fix spacing Signed-off-by: Mike Zhong <[email protected]> * remove breakpoint Signed-off-by: Mike Zhong <[email protected]> * Output ctx.working_directory as single output Signed-off-by: Mike Zhong <[email protected]> * more doc strings Signed-off-by: Mike Zhong <[email protected]> * more comments Signed-off-by: Mike Zhong <[email protected]> * Added comments Signed-off-by: Mike Zhong <[email protected]> * Add tests and test files from other branch Signed-off-by: Mike Zhong <[email protected]> * minor fix, have function return the instance rather than create. It seems to get registered when flyte packages your project Signed-off-by: Mike Zhong <[email protected]> * fix tests Signed-off-by: Mike Zhong <[email protected]> * remove set flags not supported by sh Signed-off-by: Mike Zhong <[email protected]> * fix spellcheck lint errors Signed-off-by: Mike Zhong <[email protected]> * don't have a windows equivalent test script so bypassing those tests for now Signed-off-by: Mike Zhong <[email protected]> * Add typing to make_export_string_from_env_dict params Signed-off-by: Mike Zhong <[email protected]> * fixup doc string Signed-off-by: Mike Zhong <[email protected]> * Refactored the new behavior out into a new class. Did not change implementation details at all Signed-off-by: Mike Zhong <[email protected]> * Address linter issues and up test cov Signed-off-by: Mike Zhong <[email protected]> * Skip test on windows, no equivalent script Signed-off-by: Mike Zhong <[email protected]> * Run black and isort, address SC2236 Signed-off-by: Mike Zhong <[email protected]> * Refactored class name to remove _, make utility function require name so multiple uses in a workflow don't break Signed-off-by: Mike Zhong <[email protected]> * fix utility function Signed-off-by: Mike Zhong <[email protected]> * fix utility function call in tests Signed-off-by: Mike Zhong <[email protected]> * Fix isort on test_shell.py Signed-off-by: Mike Zhong <[email protected]> * adding logging settings to papermill plugin Signed-off-by: Calvin Leather <[email protected]> * set level to info logging * improved comments/docs --------- Signed-off-by: Mike Zhong <[email protected]> Signed-off-by: Calvin Leather <[email protected]> Co-authored-by: Mike Zhong <[email protected]> Co-authored-by: Mike Zhong <[email protected]> * Set lower bound for s3fs (#1598) * set lower bound for s3fs Signed-off-by: Kevin Su <[email protected]> * update doc-requirements.txt Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Implement horovod task in mpi plugin (#1575) * Add horovod task to mpi plugin Signed-off-by: byhsu <[email protected]> * Remove unused Signed-off-by: byhsu <[email protected]> * add test Signed-off-by: byhsu <[email protected]> * inherit from mpifunctiontask Signed-off-by: byhsu <[email protected]> * fix format Signed-off-by: byhsu <[email protected]> * fix format Signed-off-by: byhsu <[email protected]> --------- Signed-off-by: byhsu <[email protected]> Co-authored-by: byhsu <[email protected]> * add ImageRender class for Flyte Decks (#1599) Signed-off-by: esad <[email protected]> * Bump tensorflow from 2.10.0 to 2.11.1 in /plugins/flytekit-mlflow (#1586) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.10.0 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](tensorflow/tensorflow@v2.10.0...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:development ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Handle batchable lists of length 0 (#1600) Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * pyflyte run imperative workflow (#1597) * pyflyte run imperative workflow Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Backfill should fill in the right input vars (#1593) * Backfill fix - Backfill was using incorrect arguments - backfill should use the argument that user provides or none at all Signed-off-by: Ketan Umare <[email protected]> * Updated code Signed-off-by: Ketan Umare <[email protected]> * fixed unit test Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> * Pyflyte prettified (#1602) * Pyflyte prettified Signed-off-by: Ketan Umare <[email protected]> * added dependency Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> * pyflyte run now supports json/yaml files (#1606) * pyflyte run now supports json files Signed-off-by: Ketan Umare <[email protected]> * added yaml support Signed-off-by: Ketan Umare <[email protected]> * fixed parsing Signed-off-by: Ketan Umare <[email protected]> * fixed windows test Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> * Add is_inside func to ImageSpec (#1601) * Add is_inside func to ImageSpec Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * Add is_inside func to ImageSpec Signed-off-by: Kevin Su <[email protected]> * Add tests Signed-off-by: Kevin Su <[email protected]> * rename Signed-off-by: Kevin Su <[email protected]> * rename Signed-off-by: Kevin Su <[email protected]> * check execution mode Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Measuring the time taken for each component when executing a task (#1581) * add time measurement framework Signed-off-by: Yicheng-Lu-llll <[email protected]> * add time measurement framework Signed-off-by: Yicheng-Lu-llll <[email protected]> * nit Signed-off-by: Yicheng-Lu-llll <[email protected]> * nit Signed-off-by: Yicheng-Lu-llll <[email protected]> * nit Signed-off-by: Yicheng-Lu-llll <[email protected]> * nit Signed-off-by: Yicheng-Lu-llll <[email protected]> * nit Signed-off-by: Yicheng-Lu-llll <[email protected]> --------- Signed-off-by: Yicheng-Lu-llll <[email protected]> * Skip the image building process if the check for its existence fails (#1614) * Fixed tests Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * add lru cache Signed-off-by: Kevin Su <[email protected]> * update Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Enable torch elastic training (torchrun) (#1603) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> This PR brings [torch elastic training (`torchrun`)](https://pytorch.org/docs/stable/elastic/run.html) to the pytorch plugin: ```python from flytekitplugins.kfpytorch import Elastic @task( task_config=Elastic( replicas=4, nproc_per_node=4, ... ), ... ) def train(...): ... ``` flyteorg/flyte#3614 * update sphinx youtube plugin (#1619) * update sphinx youtube plugin Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Read polar dataframe without copying to local (#1618) Signed-off-by: Kevin Su <[email protected]> * Add default spark image (#1616) * Add default spark image Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Lazy load modules (#1590) * lazy load module Signed-off-by: Kevin Su <[email protected]> * lazy load module Signed-off-by: Kevin Su <[email protected]> * import Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * keep structured dataset in flytekit init Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fixed tess Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fixed tests Signed-off-by: Kevin Su <[email protected]> * fixed tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * move import pandas to __init__ Signed-off-by: Kevin Su <[email protected]> * use lazy import loader instead Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * regular import Signed-off-by: Kevin Su <[email protected]> * fixed tests Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Override node name with a name containing underscore (#1608) * override node_name with a name containing underscore Signed-off-by: Kevin Su <[email protected]> * add comment Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * External Plugin Service (grpc) (#1524) External Plugin Service * Include traceback in errors from admin (#1621) Signed-off-by: Yee Hing Tong <[email protected]> * Add support for copying all the files in source root (#1622) * Add support for copying all the files in source root Signed-off-by: Kevin Su <[email protected]> * Add tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * fix PipelineModel transformer issue 3648 (#1623) Signed-off-by: esad <[email protected]> * update GH workflow for external plugin service (#1624) Signed-off-by: Kevin Su <[email protected]> * [Issue-3617] Enables FlyteFiles, FlyteDirectors, and StructuredDatasets inputs in papermill plugin (#1612) * Add support for using a list as an input for a subworkflow (#1605) Signed-off-by: Kevin Su <[email protected]> * Improve task decorator type hints with overload (#1631) Without the overload, the decorated function does not have the proper type of PythonFunctionTask, leading to spurious type errors when attempting to register the task on a FlyteRemote object Signed-off-by: Matthew Hoffman <[email protected]> * add annotation option for serialization (#1615) * add annotation option for serialization * add support for guessing type * formatting * test both directions * remove guessing for annotated types * clarify usage in test * Update tests/flytekit/unit/core/test_type_engine.py --------- Co-authored-by: Eli Bixby <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * Delete removed data persistence classes from docs (#1633) Signed-off-by: Niels Bantilan <[email protected]> * Improve workflow decorator type hints with overload (#1635) Previously, the workflow decorator is hinted as always returning a WorkflowBase, which is not true when _workflow_function is None; similar to #1631, we propose using typing.overload to differentiate the return type of workflow based on the value of _workflow_function Signed-off-by: Matthew Hoffman <[email protected]> * Fix mypy errors (#1313) * wip Signed-off-by: Kevin Su <[email protected]> * Fix mypy errors Signed-off-by: Kevin Su <[email protected]> * Fix mypy errors Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * Update type Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update dev-requirements.txt Signed-off-by: Kevin Su <[email protected]> * Address comment Signed-off-by: Kevin Su <[email protected]> * upgrade torch Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * type-protobuf<4 Signed-off-by: eduardo apolinario <[email protected]> * Allow for python 3.7 Signed-off-by: eduardo apolinario <[email protected]> * Fix protobuf transformer merge error * Fix bad merge of s/_task_models/task_models/ * Add maxsize=128 to @lru_cache to cover python 3.7 Signed-off-by: eduardo apolinario <[email protected]> * Lint Signed-off-by: eduardo apolinario <[email protected]> * Regenerate doc-requirements.txt Signed-off-by: eduardo apolinario <[email protected]> * Add missing call to lru_cache Signed-off-by: eduardo apolinario <[email protected]> * Bump version and regenerate requirements in a few plugins Signed-off-by: eduardo apolinario <[email protected]> * Revert "Fix mypy errors (#1313)" This reverts commit af49155. Signed-off-by: Eduardo Apolinario <[email protected]> * Remove docs Signed-off-by: eduardo apolinario <[email protected]> * More linting Signed-off-by: eduardo apolinario <[email protected]> * Fix type of get_type_for_output_var Signed-off-by: eduardo apolinario <[email protected]> * Remove tensorflow test_model * Replace use of shutil.copytree Signed-off-by: eduardo apolinario <[email protected]> * Other use of copytree Signed-off-by: eduardo apolinario <[email protected]> * Stringify paths in test_script_mode Signed-off-by: eduardo apolinario <[email protected]> * os.path.isfile instead in imagespec Signed-off-by: eduardo apolinario <[email protected]> * Remove tests Signed-off-by: eduardo apolinario <[email protected]> * More linting Signed-off-by: eduardo apolinario <[email protected]> * Remove flaky test Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Felix Ruess <[email protected]> Signed-off-by: franco-bocci <[email protected]> Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yicheng-Lu-llll <[email protected]> Signed-off-by: Artem Petrov <[email protected]> Signed-off-by: Mike Zhong <[email protected]> Signed-off-by: Calvin Leather <[email protected]> Signed-off-by: byhsu <[email protected]> Signed-off-by: esad <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Adrian Rumpold <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Felix Ruess <[email protected]> Co-authored-by: Franco Bocci <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: ggydush <[email protected]> Co-authored-by: ggydush-fn <[email protected]> Co-authored-by: Yicheng-Lu-llll <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Artem <[email protected]> Co-authored-by: Calvin Leather <[email protected]> Co-authored-by: Mike Zhong <[email protected]> Co-authored-by: Mike Zhong <[email protected]> Co-authored-by: ByronHsu <[email protected]> Co-authored-by: byhsu <[email protected]> Co-authored-by: peridotml <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Co-authored-by: Eli Bixby <[email protected]> Co-authored-by: Eli Bixby <[email protected]>
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
Adds a new command to pyflyte which simplifies the process of activating new launch plans
Type
Are all requirements met?
Complete description
I found that
SynchronousFlyteClienthas update_launch_plan method. So I decided to wrap it up as a CLI-command.Tracking Issue
flyteorg/flyte#3587
Follow-up issue
NA